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INTRODUCTION 

A  typical  communication  system,  as  shown  in  Fig.  1,  employs 
an  encoder  to  improve  the  efficiency  of  channel  because  an  en- 
coded message  is  less  susceptible  to  noise  in  the  channel.   A 
decoder  is  employed  at  the  receiving  end  to  recover  the 
original  message  from  the  encoded  message. 


SOURCE 


ENCODER 


CHANNEL 


DECODER 


RECEIVER 


NOISE 


Fig.  1.   A  typical  communication  system. 


The  encoding  procedures  discussed  herein  are  restricted  to 
binary  codes  only  wherein  symbols  0  and  1  form  the  code  alpha- 
bet.  A  letter,  symbol,  or  character  is  defined  as  an  individ- 
ual member  of  an  alphabet  set,  whereas  a  message  or  word  is  a 
finite  sequence  of  its  letters.   The  length  of  a  word  is  the 
number  of  letters  in  it. 

Encoding  or  enciphering  is  a  procedure  for  constructing 
words  using  a  finite  alphabet  to  represent  a  given  word  of  the 
original  language  on  a  one-to-one  basis.   The  inverse  operation 
of  assigning  original  words  to  the  encoded  words  is  called  de- 
coding or  deciphering. 

The  communication  channels  may  be  classified  as  either 
noiseless  or  deterministic.   Noiseless  channels  are  characterized 


by  one  and  only  one  nonzero  element  in  each  column  of  their 
channel  matrix.   A  noisy  channel  is  shown  in  Fig.  2. 
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Pig.  2.   A  noisy  channel  and  its  matrix. 

A  channel  matrix  with  one  and  only  one  nonzero  element  in 
each  row  characterizes  a  deterministic  channel.  A  determinis- 
tic channel  is  shown  in  Fig.  3» 
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Fig.  3'   Deterministic  channel  and  its  matrix. 

Binary  symmetric  and  binary  erasure  are  two  widely  dis- 
cussed channels,  and  are  shown  in  Figs.  l\.   and  $,    respectively. 
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Pig.  l|..   Binary  symmetric  channel  and  its  matrix. 
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Fig.  5«   Binary  erasure  channel  and  its  matrix. 
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Pig.    6.      Classes   of   codes. 


Various  classes  of  codes  are  summarized  (Abramson,  1963) 
in  Fig.  6. 

A  block  code  is  a  code  "which  maps  each  of  the  symbols  of 
a  source  alphabet  S  into  a  fixed  sequence  of  symbols  of  the 
code  alphabet  X,  as  shown  in  Table  1. 

Table  1 


Source  symbols  :      Code 

Sx  0 

So  11 

So  00 

%  X1 


A  block  code  is  said  to  be  nonsingular  if  all' the  words  of  the 
code  are  distinct. 

Table  2.   Block  code. 


Source  symbols  :     Code  (X±) 

Sx  0 

52  11 

53  00 
%  01 


A  block  code  is  uniquely  decodable  if,  and  only  if,  the  n^h  ex- 
tension of  the  code  is  nonsingular  for  every  finite  n.   The  n^*1 
extension  of  a  block  code  (Table  2)  which  maps  the  symbol  Sj[_ 
into  code  words  X-^  is  the  block  code  which  maps  the  sequence 
of  source  symbols  ( S^-]_S^2  •••  S^n)  into  sequences  of  code 
words  (XnXi2  •••  ^in)  >    as  shown  in  Table  3« 


Table  3«   Second  extension  of  block  code  (Table  2). 


Source  symbols 


Code 


Source  symbols 


Code 


s1s1 

S1S2 
SlS3 

SlSl+ 

S2S1 

S2S3 
SoS^ 


00 
Oil 
000 
001 

110 

1111 

1100 
1101 


SoS 


c3CDl 

So  So 
St  So 

33Sk 

Sjjs2 
SI4-S3 


000 
0011 
0000 
0001 

010 
0111 
0100 
0101 


A  uniquely  decodable  code  is  said  to  be  instantaneous  if  it  is 
possible  to  decode  each  word  in  a  sequence  without  reference  to 
succeeding  code  symbols,  as  shown  below. 

Received  message    0010110111 
Code  alphabet       0,  0  1,  0  1  1,  0  1  1  1 
Deciphered  message  0,  0  1,  0  1  1,  0  1  1  1 
Some  of  the  basic  binary  encoding  procedures  are  Shannon- 
Fano  coding,  Shannon  binary  encoding,  Huffman's  minimum  re- 
dundancy codes,  and  Gilbert-Moore  encoding.   These  codes  can 
neither  detect  nor  correct  an  error  introduced  during  the 
transmission  of  information  in  a  noisy  channel.   The  error- 
detecting  and  error-correcting  codes  are  designed  to  correct 
certain  patterns  of  errors. 

Shannon-Fano  Encoding 


An  ensemble  of  messages  (X)  =  (X]_,  X2,  ...,  Xn)  with  prob- 
ability (P)  =  (P]_,  P2»  •••>  Pn)  are  firsk  arranged  in  the  order 
of  decreasing  probabilities.   This  set  is  then  partitioned  into 
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two  most  nearly  equiprobable  subsets.  Then  a  "0"  is  assigned 
to  each  message  in  one  subset,  and  "1"  to  each  message  in  the 
other  subset.  The  same  procedure  is  repeated  until  each  sub- 
set contains  only  one  message  as  shown  in  Table  i|. 

Table  l±.      Shannon-Fano  code. 
Message    : Probability    : Encoded  message 

Xi  .3 

X2  .2 

X3  .2 

Xh  .15 

X?  .1 

xi  .05 


The  average  length  of  a  code  word  is  defined  as 
L  =  c-   P(Xj_)nj_,  where  n-j_  is  the  length  of  the  code  word  and 
P(Xj_)  is  the  probability  of  occurrence  of  message  X^.   It  is 
found  to  be  2.\\$   for  the  code  in  Table  Ij..   The  efficiency  of 
coding  technique  is  defined  as 

^i  =- (1) 

L  •  log  D 

where   H(X)  =  -  L—    p*  log  p. 

I  (2) 

D  =  number  of  symbols  in  the  code  alphabet 

The  source  entropy  H(X)  can  be  shown  to  be  2 . Ij.08  bits  per 

symbol.   The  efficiency  of  the  above  code  is  98.5  per  cent. 


Shannon's  Binary  Encoding 

An  ensemble  of  messages  is  written  in  order  of  decreasing 
probabilities  and  a  sequence  of  's  is  computed  as  shown  below. 


0 

2 

00 

.k 

2 

01 

.7 

3 

101 

.9 

k 

1110 

-<1  =  0 

-<2  =  P(Xi) 

-<3  =  p(x2)  +  P(x1)  =  p(x2)  +  -<2  (3) 

where  X^  Is  the  message  with  highest  probability.   A  set  of 
integers  n.j_  is  determined  using  the  inequality 

P(Xi)'^  2"ni  for  each  P(Xi)  (Ij.) 

The  -< '  s  are  then  expressed  in  the  binary  form  as  shown  in 
Table  5. 

Table  5>.   Shannon's  binary  code. 
Message  Xj_   :    Probability  P(X^)   :    -<i   :    ni   :    Code 

X2  .3 

X,  .2 

x£  .i 

For  this  case,  average  length,  the  source  entropy,  and  the  code 
efficiency  are  2.i|  bits,  1.814.6  bits  per  symbol,  and  77  per  cent, 
respectively. 

Gilbert-Moore  Encoding 

A  message  ensemble  is  written  in  the  specified  order  and 
a  set  of  integers  n^_  is  computed  using  the  following  inequality: 

21_ni  ^  P(Xi)  <22"ni  (5) 

A  sequence  of  -< '  s  is  defined  below. 

.*!  =  1/2  P(Xi) 

-<2  =  P(XX)  +  1/2  P(X2) 


±  =  P(X3_)  +  P(X2)  +  .  .  .  +  1/2  P(Xi)        (6) 


-<-?  = 
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where  X-j_  is  the  first  message,  X2  the  second,  and  so  on.   The 
-< '  s  thus  computed  are  then  written  as  binary  digits  of  length 
n^_  as  was  done  under  Shannon's  encoding  procedure  and  is  shown 
in  Table  6. 

Table  6.   Gilbert-Moore  encoding. 
Message  X±      :    Probability  P(Xj_)   :    -H   :    ni   :    Code 

X1  .1 

X2  .2 

X^  .3 

The  average  length,  source  entropy,  and  efficiency  for  this 
example  are  3>k-   bits,  I.8I4.6  bits  per  symbol,  and  £lj..2  per 
cent,  respectively. 

Huffman's  Minimum  Redundancy  Codes 

L  log  D  -  H(X) 

1  -  Efficiency  =  = (7) 

L  log  D 

This  is  the  optimum  or  minimal  redundant  encoding  procedure 

for  a  given  source  which  can  be  coded  as  shown  in  Fig.  7-   An 

optimum  code  satisfies  the  condition  L(Xn_^)  =  ^(X  ) . 

The  average  length,  source  entropy,  and  efficiency  for 

the  code  in  Fig.  7  are  3  bits,  2.972  bits  per  symbol,  and 

99.06  per  cent,  respectively. 


CODE  WORDS 
o  I 


i  i  o 

l  0   I 


0  0  0 

0  0  I 
Mil 

I    I    1  0 
I    0  0   I 

10  0  0 


55 

- 

i 

1 

i 

.45 

0 

.3 

i 

I 

1 

- 

1 

w 

.25 

0 

°s 

> 

i 

1 

.15 

.2 

0 

t 

i 

i 

T~ 

15 

0 

.15 

i 

.1 

0 

i 

i 

.1   ± 

.1   * 

.09 -1 

.06^ 

•  06-L. 

.0/1  a 

Fig.  7.   Huffman's  code. 


10 
CODING  FOR  NOISY"  CHANNEL 

Binary  codes  for  the  noiseless  channels  were  considered  in 
the  last  chapter,  whereas  those  for  noisy  channels  are  con- 
sidered here.   When  a  sequence  of  0's  and  l's  is  transmitted 
over  a  noisy  channel,  some  digits  will  be  received  in  error 
and  since  the  source  binits  are  independent,  there  is  no  way  of 
detecting  erroneous  digits.   However,  if  each  message  digit  is 
transmitted  an  odd  number  of  times  and  the  most  often  occurring 
digits  may  be  selected  at  the  receiver  in  order  to  detect  the 
original  transmitted  sequence.   For  instance,  if  each  digit  in 
0010110  is  transmitted  three  times  as  000  000  111  000  111  111 
111  000  and  the  received  message  is  100  001  101  000  110  111  Oil 
010,  the  original  message  can  be  obtained  by  selecting  the  digit 
occurring  more  frequently  in  each  sequence  of  three  digits.   It 
can  be  seen  that  000  indicates  that  the  transmitted  digit  is  0, 
whereas  the  second  sequence  111  indicates  1  as  the  transmitted 
digit,  and  so  on.   Using  this  technique,  the  original  message 
00101110  is  obtained. 

Thus  it  is  obvious  that  in  order  to  detect  and/or  to  cor- 
rect an  error,  transmission  rate  has  to  be  sacrificed,  or  in 
other  words  it  is  necessary  to  insert  redundant  bits  in  the 
message  at  the  transmitter.   In  codes  of  length  n,  m  message 
bits  are  encoded  by  inserting  k  redundant  digits  such  as 
n  =  m  +  k,  and  in  the  above  example  m  =  1,  k  =  2,  and  n  =  3. 
Codes  with  all  code  words  of  equal  length  are  usually  designated 
as  block  codes. 

Definitions  of  some  basic  terms  are  summarized  below. 
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Memoryless  Channel  is  a  channel  in  which  the  probability 
of  error  for  a  received  bit  is  independent  of  the 
receiving  bits. 

Parity  Check  Digits  in  a  message  are  used  for  detection 
and  correction  of  error. 

Even  Parity  Check  in  a  message  n  digits  long  with  the 

first  n  -  1  information  digits  and  in  the  n""1  place, 
a  0  or  1  is  assigned  so  that  the  entire  message  has 
an  even  number  of  l's. 

Odd  Parity  Check,  as  in  the  above  case,  is  an  0  or  1  in 
the  n  place,  such  that  the  entire  message  has  an 
odd  number  of  l's. 

Weight  of  a  Code  Word  is  the  number  of  l's  in  the  word. 

Group  Code  is  a  binary  code  having  the  group  property 
( see  Appendix) . 

Systematic  Code  is  a  block  code  of  length  n,  with  m  in- 
formation digits  and  k  +  n  -  m  parity  check  digits. 

Linear  Code  is  a  mathematical  term  for  n-ary  (binary, 

ternary,  etc.)  code  having  a  specific  property.   For 
binary  codes,  terms  like  linear  codes  and  group  codes 
are  synonymous. 

Hamming  Codes 

The  first  error  correcting  and  detecting  codes  are  due  to 
Hamming  (1950).   Some  details  of  these  codes  follow. 

Single  Error  Detecting  Code  (SED)  is,  for  instance,  an  n 
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bits  long  code  with  even  parity  check.   Any  single  error  would 
result  in  an  odd  number  of  l's  in  the  received  code  words.   All 
other  errors  will  go  undetected. 

Single  Error  Correction  Codes  (SEC)  can  correct  any  single 
error  occurring  within  a  code  word.   Consider  a  code  n  binits 
long,  xn  of  which  are  information  digits  and  the  remaining 
k  =  n  -  m  are  parity  check  digits.   The  values  of  the  check 
digits  are  determined  in  the  encoding  operation  by  even  parity 
checks  over  selected  information  places.   Parity  check  rules 
are  assigned  as  shown  later,  and  then  the  parity  checks  are  ap- 
plied in  such  a  way  that  every  time  a  parity  check  agrees  with 
the  value  observed  in  the  corresponding  check  digit,  a  "0"  is 
recorded  and  a  "l"  is  recorded  otherwise.   This  sequence  of 
"0's"  and  "l's",  k  digit  long  written  from  right  to  left  is  a 
binary  number  giving  the  position  of  the  error.   Since  the 
number  is  required  to  give  the  position  of  single  error  in  the 
code,  any  position  having  a  "1"  on  the  right  of  its  binary 
representation  must  cause  the  parity  check  to  fail.   So  the 
first  parity  check  must  be  over  positions  which  have  "1"  as 
the  first  digit  in  binary  numbers,  such  as  1,  3,    5,    1,    9,  ..., 
etc.   Similarly,  the  second  parity  check  should  be  over  the 
position  numbers  whose  binary  representation  has  "1"  as  their 
second  digits,  namely,  2,  3,    °,  7,    •••,  etc.   Similarly,  posi- 
tions for  the  third  parity  check  would  be  Ij.,  5,  6,  7,  etc. 
See  Table  7. 

An  illustration  of  Hamming's  SEC  code  for  n  =  7  is  discussed 
here.   Information  positions  in  the  code  shown  above  are  3,    5, 
6,  7.   If  the  corresponding  information  digits  are  10  11,  the 
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Table  7 

Parity  check  :     Location  of  : 

number  :     check  digit  :   Positions  checked 

1  1  1,3,5,7,9,  ... 

2  2  2,3,6,7,10,11,  .. 

3  k  k, 5, 6, 7, 12,  ... 

t  5  6,9,10,11,12  ... 


complete  code  word  would  be  0110011  using  even  party  checks, 
there  are  lj.  information  positions,  (2)^"  =  16  different  se- 
quences of  the  code  words  are  possible.   Suppose  the  received 
word  is  011011.   Using  the  procedure  described  above, when  first 
parity  check  is  applied  over  positions  1,  3,  5,  7,  it  is  found 
that  the  digit  in  position  1  should  be  a  1,  but  there  is  a  zero 
in  that  position.   So  the  first  digit  of  checking  number  is  1. 
Similarly,  the  digit  in  second  position  agrees  and  the  digit  in 
fourth  position  disagrees,  so  the  checking  number  is  101.   Hence 
the  error  is  in  fifth  position  as  101  is  the  binary  representa- 
tion of  5. 

Single  Error  Correcting  and  Double  Error  Detecting  Codes 
can  be  obtained  from  single  error  correction  codes  by  simply 
adding  an  additional  digit  in  the  end,  which  is  an  even  parity 
check  over  all  previous  digits.   The  operation  of  this  code  is 
summarized  in  Table  8. 

Hamming  introduced  a  geometrical  model  to  specify  the  re- 
quirements for  codes  which  can  either  detect  or  correct  more 
than  two  errors.   This  consists  of  identifying  sequences  of 
0's  and  l's  in  each  code  word  with  a  point  in  n-dimensional 
space.   For  large  n,  this  concept  is  difficult  to  visualize  but 
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Table  8 


Type  of  error 


Correction  procedure 


No  error 


Single  error 


Double  error 


More  than  two 
errors 


All  parity  checks  are  satisfied  and 
checking  number  is  000. 

Last  parity  check  fails.   Checking  num- 
ber indicates  the  position  of  error.   If 
checking  number  is  zero,  it  means  an 
error  in  the  last  check  position. 

Last  parity  check  is  satisfied  and  the 
checking  number  is  obtained  but  gives  no 
information  about  the  position  of  error. 

No  useful  information  is  obtained  and  if 
the  resulting  number  of  errors  is  odd, 
checking  number  might  create  an  addi- 
tional error. 


the  case  of  n  =  3  can  be  used  to  explain  the  basic  concept. 
For  n  =  3,  (2)3  =  8  words  can  be  associated  with  the  corners  of 
a  cube  as  shown  in  Pig.  8.   The  distance  D(U,  V)  between  two 


(001) 


Cm) 


(100) 


Cuo) 


*-  X 


Fig.  8.   Three-dimensional  cube 


15 


points,  U  and  V,  is  the  number  of  positions  in  which  they 
differ.   If  U  =  100  and  V  =  Oil,  then  D(U,  V)  =  3. 

This  shows  that  in  order  to  make  a  transition  from  100  to 
Oil,  or  vice  versa,  three  sides  of  the  cube  must  be  traversed. 
If  all  the  points  are  used  as  code  words,  the  error  cannot  be 
detected.   Now  if  the  code  words  having  a  minimum  distance  of 
2  units  are  chosen,  a  single  error  can  be  detected  because  it 
will  cause  the  code  point  to  be  moved  in  only  one  direction  to 
a  point  which  is  not  defined  as  a  code  word.   One  such  set  of 
symbols  would  be  000,  011,  101,  110. 

Also  if  the  minimum  distance  between  code  words  is  at 
least  3,  a  single  error  will  leave  the  displaced  point  nearer 
to  correct  point  than  to  any  other  code  point,  and  therefore  a 
single  error  can  be  corrected.   The  above  discussion  is  sum- 
marized in  Table  9. 


Table  9.   Hamming's  error  detecting 
and  correcting  codes. 

Minimum   :     Correction  procedure      :  Detected  :  Corrected 

distance  :                             :  : 

1  No  error  detection  or  cor-  0          0 
rection 

2  SED  1           0 

3  SEC  11 
k.                     SEC-DED  2           1 

5  DEC  or  (DEC -DEC)  2          2 

6  DEC -TED  3          2 


16 
Slepian  Group  Codes 

Slepian  (1956)  generalized  the  codes  obtained  by  Hamming 
and  Reed-Muller.  Actually  Hamming  and  Reed-Muller  codes  form 
a  subclass  of  a  larger  class  of  codes  called  Group  Codes. 

The  encoding  scheme  for  these  codes  is  simple  because  last 
K  positions  in  the  code  word  are  assigned  parity  checks  and  the 
decoder,  which  is  a  maximum  likelihood  detector,  is  optimum 
theoretically. 

The  mathematical  definition  of  group  is  given  in  the 
appendix.   A  collection  of  binary  words  is  said  to  form  a  group 
if  the  product  of  any  two  members  is  also  a  member  of  this  col- 
lection, and  this  collection  contains  the  identity  element.   As 
an  example,  the  2n  possible  sequences  of  n  bits  form  a  binary 
group.   This  group  is  denoted  by  Bn  and  contains  2n  words  or 
elements.   An  n-place  group  code  is  defined  to  be  a  collection 
of  2m(m  «==:  n)  binit.   Code  words  that  form  a  group  as  defined 
above.   All  such  group  codes  are  subgroups  of  Bn  which  contains 
all  2n  possible  sequences  of  n  binits. 

According  to  Slepian  there  are  exactly 

(2n  -  2°)(2n  -  21)  ...  (2n  -  2m"1) 

N(n,m)  = —       (8) 

(2m  -  2°)(2m  -  21)  ...  (2m  -  2m"1) 

different  subgroups  of  Bn  having  2m  elements.   And  these  N(n,m) 
possible  codes  are  tabulated  by  Slepian  (1956)  and  many  ex- 
amples of  such  codes  are  also  given. 

After  choosing  a  (n,m)  code  and  2m  words,  the  information 
is  transmitted  by  selecting  blocks  of  m  message  digits  and 
associating  these  in  a  one-to-one  manner  with  the  2m  code  words 


•  • 
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Since  the  channel  Is  noisy,  some  digits  in  the  received  code 
word  will  be  in  error.   These  errors  are  then  corrected  knowing 
that  transmitted  words  formed  a  group. 

Slepian  has  suggested  an  optimum  decoding  method  which 
uses  a  standard  array  of  code  words,  which  is  obtained  as  fol- 
lows.  The  words  of  a  specific  code  word  A^  =  I  =  000  ..  0, 
A2,  A^  ...  Am  are  placed  in  the  top  row  and  the  words  S2,  So 
Sv  are  chosen  such  that  S2  is  any  code  word  hot  contained  in 
the  first  row.   S^  is  any  code  word  not  contained  in  the  first 
two  rows  and  so  on.   The  whole  array  then  can  be  developed  as 
shown  below. 

I    A2    A3  A^ 

S2   S2A2   S2A3 S2A^ 

B    =    S3   S3A2   S3A3  ......  3^ 


Sv   SVA2   SVA3 S^ 

All  the  rows  except  the  first  are  called  Cosets  and  the 
first  words  S2,  S3,  ...,  Sv  are  called  Coset  Leaders.   The 
weight  W-j_  a   of  an  element  in  the  array  is  the  number  of  nonzero 
digits  in  the  n  binit  word  located  in  i-"1  row  and  j    column. 
If  the  coset  leaders  are  chosen  as  the  elements  with  the  minimum 
weight,  such  an  array  is  called  Standard  Array.   Peterson  (19&1) 
gives  several  theorems  using  the  properties  of  standard  array 
coset  and  coset  leaders.   Slepian  (19£6)  and  Prange  (1957) 
offered  a  theorem  using  properties  of  coset  and  coset  leader 
for  detection  of  group  codes.   The  decoding  process  is  illus- 
trated with  a  .specific  example  as  given  below. 
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A  standard  array  for  (i|,2)  code  0000,  1100,  0011,  1111  is 
as  follows: 


0000 

1100 

0011 

1111 

0001 

1101 

0010 

1110 

0100 

1000 

0111 

1011 

0110 

1010 

0101 

1001 

It  is  formed  by  the  method  illustrated  in  the  preceding  para- 
graph.  Note  that  many  standard  arrays  can  be  formed  by  choos- 
ing different  coset  leaders  having  the  same  "weight.   Consider- 
ing a  BSC,  when  a  word,  say  A,  is  sent,  the  word  received  can 
be  any  element  of  Bn.   If  the  received  word  lies  in  the  i^n 
column,  the  detector  will  show  that  word  Aj_  has  been  trans- 
mitted.  In  the  above  example,  0111  will  be  produced  as  0011, 
and  1010  will  be  produced  as  1100.   Moreover,  considering  the 
fact  that  any  word  in  a  standard  array  is  at  least  as  close  to 
the  code  words  at  the  top  of  its  column  as  it  is  to  any  other 
transmitted  code  word,  this  scheme  gives  maximum  likelihood  de- 
tection that  is  the  detected  symbol  is  the  one  most  likely  to 
have  been  transmitted.   The  only  drawback  of  this  decoding 
scheme  is  that  it  requires  knowledge  of  all  2n  possible  words. 
Hence  the  detection  equipment  grows  exponentially  with  the 
length  of  the  code  word.   To  overcome  this  disadvantage,  a 
simple  encoding  procedure  for  the  generation  of  code  words  by 
parity  checks  is  given  in  the  following  example.   Definitions 
of  Systematic  Code  and  Equivalent  Code  are  also  given  below. 
In  a  Systematic  Code,  all  code  words  have  the  same  in- 
formation digit  locations  and  same  parity  check 
digit  locations. 
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Two  group  codes  are  Equivalent  if  one  can  be  obtained 
from  the  other  by  permitting  the  digit  locations. 
The  generation  of  groups  codes  by  parity  check  is 
explained  in  the  following  example. 
Consider  the  same  (l+,2)  code  given  by  0000,  1100,  0011, 
1111.   Positions  2  and  3  &re   assumed  to  be  information  posi- 
tions.  If  a  code  word  is  denoted  as  X1X2X3X],,  the  parity  check 
rules  are  determined  by  solving  the  simultaneous  equations 

Xl  =  A1X2  ®  A2X3  (10) 

\  =   A3X2  e  AI|.X3  (11) 

(Note  that  ©  means  addition  modulo  2.)   Substituting  values 

from  third  and  fourth  code  words, 

1  =  Ax  .  0  ©  A2I    A2  =  1 
1  =  Ax  •  1  ©  A21    Ai  =  0 

1  =  A3  •  0  ©  Ajjl    Au    =  1 
1  =  A3  .  1  ©  Ajl    A3  =  0 

so  A2  =  A[^_  =  1  and  A]_  =  A3  =  0. 

Thus  the  parity  check  rules  are  X]_  =  X3  and  Xl    =   X3. 

Slepian  (195&)  has  given  two  important  results  involving 
systematic  and  equivalent  code. 

1.  Every  group  code  is  a  systematic  code  and  vice  versa. 

2.  Every  (n,m)  code  is  equivalent  to  a  (n,m)    code  having 

first  m  places  as  information  digits  and  remaining  k  =  n  -  m 

places  as  parity  check  over  the  first  m  places.   The  general 

expression  for  k  check  digits  then  becomes 

m 
Xj_  =  ^2"  viixi  wnere  i  =  m  +  1,  ...,  n       (12) 

3=1 

(Summation  is  modulo  2  with  the  multiplication  rules  being 
0.1=1.0=0-0=0,  and  1  •  1=1.)   Slepian  also  lists 
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suitable  parity  check  rules  for  optimum  results.  For  instance, 
consider  a  (6,3)  code.  In  this  case  number  of  words  =  (2)3  =  8 
and  parity  check  rules  are: 

X^  =  X-l  ©  X2  (13) 

X£  =  Xx  ©  X3  (11+) 

X6  =  X2  ©  x3  (15) 

resulting  in  Table  10. 

Table  10 

Code  words       Information     :     Parity  check 
:    digits         :         J 

1  000  0   0   0 

2  001  0    11 

3  010  10    1 
h,  011  110 

5  100  110 

6  101  10    1 

7  110  Oil 

8  111  0    0    0 

Slepian  has  described  a  method  of  maximum  likelihood  de- 
tection by  means  of  parity  checks.   In  this  method  for  a 
specific  (n,m)  code,  parity  check  rules  are  applied  to  any  code 
word,  say  T,  and  check  digits  resulting  from  the  i^*1   parity 
check  are  compared  with  the  digit  in  2nd  position  of  T .   If 
these  agree,  a  "0"  is  assigned;  otherwise  a  "1"  is  assigned. 
In  this  way  a  parity  check  sequence  denoted  by  R(T)  is  thus 
formed  by  writing  "1"  and  "0"  from  left  to  right. 

If  the  message  received  is  101001  =  T,  and  the  first  three 
bits  are  information  digits,  namely,  101,  then  the  parity  check 
word  for  this  code  is  001,  while  (from  Table  l±)    it  should  be 
101.   Since  i|th  parity  check  is  different  and  5th  and  6th  parity 
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checks  are  the  same,  the  checking  number  is  100  =  R(T) ,  and 
therefore  the  error  is  in  the  i|th  position. 

Using  the  parity  check  rules  of  the  last  example, 

Let  T  =  (101001) 

Xx  +  X2  =  X^   so  assign  a  "1" 

X]_  +  Xo  =  Xt      so  assign  a  0 

X2  +  X3  =  X6   so  assiSn  a  "°" 
R(T)  =  (100) 

Slepian  has  proved  the  following  theorem  using  this  definition 
of  R(T) .   If  I,  A2,  A,,  ...,  A^  is  a  (n,m)  code  and  consider 
Bn  to  be  developed  in  this  standard  array  by  this  code.   Let 
R(T)  be  the  parity  check  sequence  for  a  word  T  which  has  been 
formed  in  accordance  with  the  parity  check  rules  of  the  speci- 
fied code.   Then  R(Tq_)  =  R(T2)  if  and  only  if  T1  and  T2  lie  in 
the  same  row  of  the  standard  array. 
Consider  (ij.,2)  code 


0000 

1011 

0101 

1110 

0010 

1001 

0111 

1100 

0100 

1111 

0001 

1010 

1000 

0011 

1101 

0110 

Parity  check  rules  are  Xo  =  X^  and  Xl  =  X]_  ©  X2. 

By  definition  all  words  in  the  first  row  satisfy  parity 
check  giving  a  parity  check  sequence  of  00.   In  the  second  row, 
every  word  fails  the  first  parity  check  and  satisfies  the  second 
check.   Therefore  the  parity  check  sequence  for  this  row  is  10. 
Similarly,  parity  check  sequences  for  third  and  fourth  row  are 
01  and  11,  respectively.   So  one  can  establish  the  following 
correspondence  between  the  parity  check  sequence  and  coset 
leaders. 
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00  — >  Si  =  0000 

10  — >  s2  =  0010 

01 ->  S3  =  0100 

11 ^Sl  =  0001 

Maximum  likelihood  detection  is  now  obtained  as  described  in 
the  following  paragraph. 

Parity  check  sequence  R(T)  of  the  received  code  word  T  is 
formed,  which  in  turn  identifies  the  coset  leader,  say  S^_.   The 
product  S^  T  is  produced  as  the  detector  output.   Suppose 
(0001)  of  (i|,2)  code  is  received.   R(0001)  =  01  wherein 
corresponds  to  So  =  0100.   The  detected  word  is  therefore 
(0100)  (0001)  =  0101. 

Some  of  the  recent  and  advanced  block  coding  techniques 
(Peterson,  19&1)  are  discussed  in  this  section.   Some  pertinent 
definitions  are  given  here  to  supplement  the  material  in  the 
appendix. 

Bose-Chaudhuri  Codes 

These  cyclic  codes  are  defined  in  terms  of  the  roots  of  a 
generator  polynomial.   A  generator  polynomial  g(x)  is  defined 
as  the  monic  polynomial  of  the  smallest  degree,  such  that  g(x) 
is  in  an  ideal  I  (see  appendix  for  definition  of  an  Ideal). 

Assume  the  symbols  to  be  elements  of  Galois  field  of  q 
elements,  i.e.,  GF(q).   Let  m  be  an  integer  and  L  be  any  ele- 
ment of  GF(qin).   Then  the  code  consisting  of  all  vectors  (f(x)) 
over  GF(q)  for  which  -<m0,  *xn0+1,  ...,  ^™0+d-2  ape  roots  of  f(x) 

is  a  Bose-Chaudhuri  code.   For  the  binary  case   q  =  2  and  let 
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itiq  =  1  and  d  =  2^+l  where'  t  is  any  integer.   (f(x))  is  a  code 
vector  if  and  only  if  ^1,  u2 ,  .<3^  #  t  #  f    ^2t  are  roo-ts  0f  f(x). 
The  minimum  function  of  any  element  B  in  the  extension  field 
(see  appendix)  is  defined  as  the  monic  polynomial  of  the  small- 
est degree  with  the  coefficients  in  the  ground  field  F.   Let 
m]_(x)  denote  the  minimum  function  of  l1  where  1  is  a  primitive 
element  of  GF(qm)  then  jt,    «<  ,    ^ ,    .  ..,  are  roots  of  m-j_(x)  = 
m2(x)  ...  i%(x)  =  ms(x)  ...,  -<J5,  «<  ,  -<    ...  are  roots  of 
m-a(x)  =  m£)(x),  and  jy t    -<10  are  roots  of  ...  m^(x)  =  m^o(x)  ...  . 

Hence  in  other  words,  all  the  even  powers  of  -<  are  a  root 
of  the  same  minimum  function  of  some  previous  odd  powers  of  «<, 
i.e.,  «<  ,  -o~,  -<  ,  .  ..,  are  roots  of  m-^x);  «<  ,  -<   are  roots  of 

mo(x),  and  -<    is  a  root  of  m^(x).   Therefore  it  may  be  said 

....  r>  2t-l 

that  (f(x))  is  a  code  vector  if  and  only  if  -<,  -<  ,  •••,  •< 

are  roots  of  f(x)  where  t  is  any  integer.   Therefore  the  gen- 
erator of  code  is  the  Least  Common  Multiple  (L  C  M)  of 
(m1(x)  1113  (x)  ...  m2t-l(x)). 

The  following  example  illustrates  the  procedure  for  con- 
structing a  Bose-Chaudhuri  Code.  Let  q  =  2  and  /  be  a  primi- 
tive element  of  GF(2^).   Then  j^>  =   1. 

•c1,  u.^ ,  -<^,  ^°  are  roots  of  m^(x)  =  m2(x)  =  m[,(x)  =  ms(x)  degree  J4. 
•o-5,  «<  ,  ^x  ,  j?   are  roots  of  m3(x)  =  m^(x)  degree  I4. 

-<  ,  -<  are  roots  of  m^(x)  =  m^otx)  degree  2 

Therefore    g(x)  =  m^(x)m^(x)mc(x)  (16) 

If  ^  is  a  root  of  x^-  +  x  +  1  (see  Table  11  in  Appendix),  then 
m]_(x)  =  x*+  +  x  +  1.  nv^x)  and  iru;(x)  are  calculated  as  shown. 
Let    1113  (x)  =  a0  +  a-^  +  a2x2  +  a3X^  +  x^  (17) 


2k 


Substitute 


Then 


a0 


,3  . 


(0001)  for  x 


J>  =  (0011)  for  x2 
-<9  =  (0101)  for  x3 
-<12=  (1111)  for  x^ 


1 

0 
0 

+  fll 

V 
0 
0 

+    &2 

0 

i 

+   a3 

roi 
i 

0 

+  ai+ 

0 

1 

I     J 

M 

. 

1 

' 

ll 

1 1 
1 


=  0 


or  aQ  +  1  =  0 

a3  +  1  =  0 

a2  +  1  =  0 

a-[_  +  a2  +  83  +  1  =  0 


Therefore 


aQ  =  1 

a3  =  1 

ao  =  1 


*!  =   1 


Therefore 
Similarly 
Therefore 


iri3(x)  =  1  +  x  +  x  +  x-*  + 


A 


m^(x)  can  be  found  to  be  =  ( x2  +  x  +  1) 
g(x)  =  (1  +  x  +  x^")  (1  +  x  +  x  +  x^  +  x^) 
(1  +  x  +  x2) 
=  1  +  x  +  x2  +  x^  +  x^  +  x°  +  x10 


(18) 


(19) 


The  multiplication  process    is    illustrated    in   the  Appendix.      The 
degree    of   g(x)    =   10    and    therefore    the   dimension   of   the   code    is 
15   -    10   =  5,    which  corresponds    to    the    information   digits.      The 
elements    are 

(g(x)j     =1110110010100000 

(xg(x))     =01110    11001010000 

(x2g(x))     =0011101100101000 

(x3g(x))     =0001110110010100 

(x^g(x))     =0000111011001010 

The   generator  matrix   is 
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1110110010100000 
0111011001010000 

0011101100101000  (20) 

0001110110010100 
0000111011001010 
Bose   and   Chaudhuri  have    summarized   all    the   codes   generated 
by  primitive    elements    of   order  less    than  2°    (Peterson,    196l) . 


Reed-Muller  Codes 


These  codes  are  a  subclass  of  Slepian  codes  and  cover  a 
wide  range  of  rate  and  minimum  distance.   These  are  different 
from  Slepian  codes  in  that  a  specific  procedure  is  available 
for  determining  a  set  of  code  words  when  the  following  rela- 
tions hold: 

n  =  2m 


,xn,  ,m, 

k  =  1  +  (1)  +  ...  +  (r) 

m, 


m 


n-k=l+  (J)  +  ...  +  (  ^  ) 


(21) 
(22) 
(23) 
(2k) 


d  =  2rn~r  =  minimum  weight 

r  =  order  of  the  code 
Consider  the  set  of  vectors  Vq,  V^,  V2,  • • • >  ^m>  over  ^he 
field  of  two  elements.   Let  V  be  the  vector  whose  2m  components 
are  all  l's  and  V-]_,  V2,  •  ••,  Vm  be  the  rows  of  a  matrix  that 
has  all  possible  m- tuples,  i.e.,  1.2  ..  2m  as  columns.   Define 
the  vector  product  of  two  vectors  as  follows. 

U  =  {a±,    a2,  83,  ,  an)  (2£) 

V  =  (bx,  b2,  b3,  ...,  bn)  (26) 

(27) 


UV  =  (a]^,  a2b2,  83^3,  ...,  anbn) 
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The  collection  of  vectors  formed  by  multiplying  vectors  Vj_  two 
at  a  time,  three  at  a  time,  ...  m  at  a  time,  gives  us  the  basis 
for  Reed-Muller  codes. 

Consider  the  case  of  m  =  3,  r  =  1.   Then  length  of  code 
n  =  23  =  8.   k  =  1  +  (3)  =  ij.. 
This  is  an  (8,1|)  code. 

Vq  =  (11111111) 
V3  =  (0  0  0  0  1  1  1  1) 
V2  =  (0  0  1  1  0  0  1  1) 
Vi  =  (0  1  0  1  0  1  0  1) 
v3v2  =(00000011) 
V^  =(0000010  1) 
V2V!  =(0001000  1) 
VJV2V2.   =  (00000001) 
These  eight  vectors  are  then  the  basis  of  a  first-order  Reed- 
Muller  code.   In  a  way  these  are  equivalent  to  the  generator 
matrix  as  in  the  Bose-Chaudhuri  case. 

Fire  Codes 

These  codes  with  symbols  from  GF(q)  are  best  defined  in 
terms  of  the  generator  polynomial. 

g(x)  =  p(x)(xc  -  1)  (28) 

where  p(x)  is  an  irreducible  polynomial  of  degree  m  over  GF(q) 
whose  roots  have  order  c,  and  is  not  divisible  by  c.   The  num- 
ber of  parity  checks  is  ( c  +  m) ,  and  therefore  the  number  of 
information  symbols  k  =  n  -  c  -  m. 

For  instance,  consider  the  fire  code  generated  by 
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(x^  +  x  +  Dix1     +  1)  .   In  this,  m  =  7,  c  =  16,  order  of 
roots  =  27  -  l  =  127,  and  n  =  16  x  127  =  2032.   Parity  check 
symbols  =c+m=7+l6=23.   Information  symbols  =  2032  -  23 
=  2009. 

Fire  codes  are  capable  of  correcting  any  single  burst  of 
length  b  ^  m  or  less,  and  of  simultaneously  detecting  any  burst 
of  length  d  >b  or  less  if  c^  b  +  d  -  1.   In  the  example  above, 
the  code  can  correct  all  bursts  of  length  7  or  less  and  simul- 
taneously detect  all  bursts  of  length  10  or  less.   It  can  detect 
any  burst  of  length  23  or  less  and  the  combination  of  two  bursts 
in  which  the  shorter  has  length  no  greater  than  7  and  the  longer 
has  length  no  greater  than  17. 

The  procedure  for  constructing  a  code  is  illustrated  below. 
Let  g(x)  =  (x3  +  x2  +  l)(x^  +  1)  (29) 

Then  m  =  3,  c  =  5-   Order  of  roots  of  x3  +  x2  +  1  =  23  -  1  =  7. 
Therefore  length  n  =  7  x  5  =  2>5>      Parity  check  symbols  =  c  +  m 
=5+3=8.   Information  symbols  =  35  -  8  =  27. 


g(x)    =   1  +  x2  +   x3   +  x^  +  x?   +   : 


8 

35th   digit 


f 

(g(x))  =    (10    110    10    110    0   0 0) 

(xg(x))  =(010110    10    1100    0) 

(x2g(xj)  =(001011010110  0) 

.  . 

(x?g(x))  =(000000010110101    1...6) 
Therefore   the    generator  matrix  G   is    then 
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G   = 


101101011000 
010110101100 
001011010110 


o 


35th  digit 


0 
0 


(30) 


00000001011010  0 

This  fire  code  is  capable  of  correcting  a  burst  of  length 
3  or  less  and  simultaneously  detecting  a  burst  of  length  3  or 
less.   For  detection  alone  it  can  detect  a  burst  of  length  no 
greater  than  8  and  a  combination  of  two  bursts  in  which  the 
shorter  burst  is  no  greater  than  3  and  the  longer  burst  is  no 
greater  than  6. 


Reed-Solomon  Codes 


These  codes  are  capable  of  correcting  more  than  one  burst 
of  error.   Peterson  defines  this  as  a  code  with  symbols  in  GF(qr) 
that  corrects  all  bursts  of  t  or  fewer  errors  on  a  channel. 
The  elements  of  GF(qr)  are  encoded  into  blocks  of  r  symbols  of 
GF(q).   Then  the  burst  of  length  (t  -  l)r  +  1  or  less  can  af- 
fect at  most  t  excessive  blocks,  and  hence  can  be  corrected. 

For  example,  consider  a  Reed-Solomon  code  with  symbols  from 
GF(2?)  and  t  =  1}.,  r  =  7,  t  =  l+.   This  could  correct  all  bursts 
of  length  (t  -  l)r  +  1,  i.e.,  (l\.   -  1)7  +  1  =  22  or  less.   The 
number  of  blocks  required  =  2  x  1+   =  8,  the  number  of  parity 
checks  =8x7=  5&,  and  the  length  of  code  is  7  x  2'  -  1  = 
7  x  127  =  889.   The  length  of  Fire  code  to  correct  the  same 
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burst  of  length,  i.e.,  22,  would  be  ,?22   -  1)  y   I4.3  =  160  mil- 
lion binary  digits.   Thus  it  is  clear  that  the  length  of  Reed- 
Solomon  code  compared  with  the  Pire  code  will  be  much  less  for 
correcting  the  burst  of  the  same  length. 

Summa  ry 

Hancock  (1962)  has  summarized  the  relationship  between  the 
various  encoding  procedures  considered  in  the  previous  sections 
b;y  means  of  a  block  diagram,  shown  in  Fig.  9.   He  compares  the 
various  encoding  procedures  for  the  noisy  channels  giving  the 
advantages  and  disadvantages  of  each. 

BIT  LOSS  AND  GAIN  CORRECTION  CODE 

The  first  single  bit  loss  or  gain  correction  code  has  been 
suggested  by  Sellers  (1962).   In  digital  communication  systems 
a  message  may  be  received  with  a  different  number  of  bits  than 
those  in  the  transmitted  message.   In  other  words,  a  message 
may  gain  or  lose  bits  due  to  the  loss  of  synchronization  between 
the  transmitter  and  the  receiver  among  other  causes.   The  syn- 
chronization systems  may  be  classified,  (Wier,  196l) ,  as  syn- 
chronous and  asynchronous.   In  the  synchronous  system,  each 
data  bit  takes  the  same  time  for  transmission,  while  in  asyn- 

conous  system  data  bits  may  take  unequal  times  for  transmis- 
sion.  In  the  synchronous  system  a  clock  at  the  transmitter  times 
the  bits  and  the  blocks  rior  to  transmission  and  thus  determiner. 
the  •  t.  of  the  dats  sou  *ce.   At  th<  receii  r  end  a  c]  ck  or  an 
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CODING 
TECHNIQUES 


COOING    FOR 
NON-BINARY 
SYSTEMS 


NOT    DISCUSSED 


CODING  WITH 
EQUAL   COST 
SYMBOLS 


SHANNON- 
!  FANO 


GILBERT   MOORE 


HUFFMAN 


SHANNONS    BINARY 


CODING  FOR 
BINARY 
SYSTEMS 


CODING  FOR 
NOISELESS 
CHANNEL 


CODING    FOR 

NOISY 

CHANNEL 


CODING   WITH 
NON-EQUAL  COST 
SYMBOLS 


JL 


BLACKMAN 


KARP 


CODING   FOR   BSC 

ORBEC  WITHOUT 

MEMORY 


MARCUS 


CODING    FOR 
CHANNEL  WITH 
MEMORY 


FIRE 


REED-SOLOMON 


GROUP 
CODES 
(SLE.PIAN) 


NOT  DISCUSSED 


HAMMING 


1 


BOSE-CHAUDHRI 


1 


SEQUENTIAL 

CODING 
(WOHENCROFT) 


NOT  DISCUSSED 


REED-MULLER 


Fig.    9.      Summary   of  relationship   between 
various    encoding  procedures. 
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oscillator  pulled  into  proper  phase,  or  a  shock-excited  ringing 
circuit  of  proper  phase  is  used  to  record  each  data  bit  once 
and  only  once. 

When  a  message  is  received  with  more  or  fewer  bits  than 
were  transmitted,  it  may  be  corrected  by  approximately  locating 
the  error  and  removing  or  inserting  bits,  respectively.   The 
approximate  location  of  the  bit  loss  or  gain  is  found  by  the 
use  of  a  special  character  which  is  inserted  into  the  code  at 
periodic  intervals.   The  special  character  has  the  property 
that  when  shifted  left  or  right  or  when  a  bit  is  gained  or  lost 
in  it,  it  gives  an  indication  of  what  has  happened  and  thus 
determines  the  corrective  action. 

Special  character  001  will  be  used  for  the  following  two 
examples  which  indicate  the  procedure  for  the  case  of  a  single 
bit  loss  or  gain.   Let  the  encoded  message  be  001a2,  a2i-l>  •••; 
a  .+-j_001  a  i    ...  a2a-j_,  where  a^  is  the  first  bit  to  be  sent  into 
the  channel.   On  receipt  of  a  count  j  bits  from  an,  one  ex- 
amines for  the  special  character. 

The  length  is  corrected  accordingly  if  a  bit  is  lost  or 
gained.   It  is  possible  that  either  an  incorrect  bit  may  be  in- 
serted or  removed,  or  a  bit  may  be  inserted  or  removed  from  the 
wrong  position.   This  results  in  a  burst  of  errors,  which  is 
then  corrected  by  a  burst  error  correcting  code. 

The  size  of  burst  of  errors  is  limited  by  the  fact  that  in 
order  to  correct  any  burst  of  length  b  or  less,  a  linear  code 
must  have  at  least  2b  parity  check  symbols  (Peterson,  196l) . 

It  can  be  generalized  as  follows. 
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J  =  length  of  block   b  =  length  of  burst  possible 

J  +  1 
1  1  =  


2 

2  2  =  J/2  +  1,  or  1  =  J  -  1 

J  +  1         J  -  1 

3  3  =  J,  or  2  =  ,  or  1  =  

2  2 

J  J         J-2 

If  If  =  J,  or  3  =  —  +  1*  or   2  =  — ,  or  1  =  

2  2  2 

J-l        J+l         J-l 

5  5  =  J>  or  i+  =  or  3  =  ,  or  2  =  

2  2  2 

J-3 
or  1  =  


2 
In  the  case  of  even  J  the  only  terra  occurring  for  all  even 
values  is  J/2  +  1  and  it  is  also  the  maximum  length  of  burst 
which  can  be  corrected,  according  to  the 'theorem  stated  above'. 
So 

b<C  J/2  +  1  (3D 

Similarly,  for  odd  J  the  only  term  occurring  for  all  odd  values 
J  +  l 

is  and  it  is  also  the  maximum  length  of  burst  of  errors 

2 
which  can  be  corrected.   So 

J  +  l 

b^  (32) 

2 

An  additive  error  is  defined  as  a  change  of  "1"  to  "0"  or 

vice  versa.   When  the  length  correction  is  applied  by  examining 

just  one  special  character  without  reference  to  any  other  special 

character,  the  process  is  called  independent  decoding.   This  is 

not  capable  of  correcting  an  additive  error  because  if  such  an 

error  is  allowed,  the  error  may  occur  in  the  special  character 
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and  thus  result  in  a  wrong  correction  procedure.   On  the  other 
hand,  if  dependent  decoding  which  includes  checking  two  con- 
secutive special  characters  and  the  information  contained  in 
these  characters  is  used  for  taking  the  corrective  action,  the 
additive  errors  can  be  corrected.   In  this  case  also 


b  <  J/2  +  1 

J  +  1 
b^  


for  j  even 
for  j  odd 


First  case 


< 

Shifted  left 


? 

Shifted  right' 


Additive  errors 


Independent  decoding 
0   0   1 
0 


0      10 
0   1°      X      X 


->  Gain  of   a   bit 


10      0 


->  Loss    of  a   bit 


10      1 


0  0  0 
110 
111 


->  fLoss   of  a   bit 

physically  in  error 
in  special  char- 
acter 


->  Impossible 


Consider  an  example  of  J  =  5>«   Let  the  encoded  message  be 

11110011100100111011 
i 1  i 1 

and  the  message  received  be 

1100111001001110111 
i t  i i 

whereas  the  special  character  is  received  as  Oil. 

There  is  gain  of  one  bit  and  so  one  bit  has  to  be  removed. 
The  requirement  is  to  remove  that  bit  such  that  if  it  causes  a 
burst  of  errors,  it  should  not  exceed  the  permissible  length  of 


314- 

5  +  i 

burst  of  error,  in  this  particular  case  =  =  3- 

2 

Hagelberger  (1959)  defines  the  length  of  a  burst  as  the 
length  of  a  binary  word  which  .describes  the  error  pattern  and 
is  formed  by  putting  a  "1"  for  each  erroneous  digit  and  a  "0" 
for  each  correct  digit.   One  essential  requirement  is  that  the 
first  and  last  digit  of  such  a  binary  word  be  "1"  because 
there  is  no  point  in  including  correct  digits  which  are  outside 
the  burst. 

So  if  a,  is  removed  and  it  may  be  incorrect  and  if  sl  and 
a^  are  also  in  error,  we  will  get  a  burst  of  length  5  which 
cannot  be  corrected.   If  a^  is  removed  no  matter  where  the  burst 
started  in  the  block,  the  burst  will  never  exceed  the  length  3 
and  so  can  always  be  corrected.   It  is  shown  below  that  for  all 
odd  J  and  for  a  single  bit  gain,  the  bit  to  be  removed  would  be 
a(J+l/2)  and  similarly  for  even  J,  a(j/2+l)  ■   Similarly,  if  a 
loss  of  a  bit  occurs,  then  the  message  received  will  be 
11111001110010011101 

L I  i L 

The  rule  governing  the  place  to  insert  the  bit  is  also 
formulated  in  accordance  with  the  theorem  on  the  size  of  burst. 
In  this  case  the  only  appropriate  place  to  insert  a  bit  is  be- 
tween a2  and  ao.   If  the  new  bit  which  would  be  in  position  a-^ 
is  in  error,  the  burst  still  cannot  exceed  a  length  of  3,  and 
hence  can  be  corrected.   It  can  be  generalized  that  a  bit  be 

inserted  between  a(j/2-l/2)  and  a(j/2+l/2)  ^or  a-^  oc^  ^* 

Similarly  for  even  J,  the  bit  will  have  to  be  inserted  between 

a (J/2-1)  and  aj/2- 

The  rules  for  dependent  decoding  are  essentially  the  same 
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except  as  stated  above.   Additive  errors  can  be  corrected  and 
also  the  information  contained  in  two  consecutive  special  char- 
acters is  used  to  determine  the  correction  procedure. 


Summary  of  the  Above  Discussion  (Sellers,  1962) 


Special 

character 

received 


Correction  procedure 


001 


No  error  or  error 
in  special  char- 
acter 


Do  nothing  this  time;  (if 
there  is  an  error,  it  will 
be  caught  by  next  special 
character) . 


010 
Oil 


Gain  of  a  bit 


Take  out  received  bit 

a(j/2+l)  ^or  ^  even 

J  +  1 

a( )  for  J  odd 


100 
101 


Loss  of  a  bit 


Insert  a  bit  between 
a( J/2-1)  and  aj/2  for  J 
even.   a(j/2-l/2)  and 
a( J/2+1/2)  for  J  odd- 


101 


Gain  or  loss  in 
special  character 


Do  nothing  this  time.   It 
will  be  caught  by  the  next 
special,  character. 


000 
110 
111 


Additive  errors 


Impossible 
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After  a 


k 


After  a 


k+1 


< 

Shifted  left 


0  0  1 


Shifted  right 


Additive  errors 


0 

0    10 
Oil 

0    1  X 

0 

10   0 
0   0   0 

1 

1     ] 

X  0   0 

0    0   0 
0    10 
Oil 
10    0 
10    1 

111 

0    1  X 

X  0   0 

Gain  of  1  bit 


Loss  of  a  bit 


Gain  of  1  bit 

and  error  in  special 

character 


Loss  of  1  bit  and 
error  in  special 
character 


The  correction  procedure  is  essentially  the  same  as  for 
independent  decoding. 
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Summary  of  Above  Discussion  (Sellers,  1962) 


Special  character 
After  a^-  :  After  a^+i 


Diagnosis    .  Correction  procedure 


001       Any  error     An  error  oc-    Wait  for  next  special 

curred  between 


100 


111 


a^  and  a^+l     character 


01X         01X         Gain  of  a  bit    Take  out  received  bit. 

ak-( J/2+1)  for  J  even 


ak-(j/2+l/2)  for  J  odd 


000  01X         Error  in  Take  out  a  bit  in 

100  special  char-  special  character 

101  acter  and  bit  after  a^. 
Ill  gained 


X00         X00         A  bit  lost      Add  a  bit  between 

ak-( J/2-1)  and 


ak-j/2  f°r  even. 
ak-( J/2-1/2)  and 
8k- (J/2-1/2)  for 


J  odd. 


Oil  X00         Error  in  Insert  a  bit  in 

010  special  char-  special  character 

100  acter  and  after  a^ 

101  bit  lost 


K  =  J  2  J 
X  =  0  or  1 
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The  Special  Character 

The  length  of  a  special  character  for  multiple  bit  loss 
arid  gain  should  be  greater  than  or  equal  to  2b  +  1,  where  b  is 
the  burst  of  length  to  be  corrected,  and  for  multiple  bit  loss 
or  gain  it  should  be  greater  than  or  equal  to  b  +  1. 

Take  a  synchronizing  character  of  length  2b,  say  0001.   It 

can  be  written  as  two  adjacent  characters  of  length  b  each. 

Denote  the  left  character  as  L '  and  right  character  as  L",  i.e., 

0  0   0  1. 

Let  the  message  transmitted  be 

L'  |  L"  | 
01000111011 

Note  that  the  information  following  the  special  character  is  L". 

If  a  loss  of  2  bits  occurs,  the  message  received  would  be 

L"   L'  | 
010001011 

showing  that  L"L'  has  replaced  L'L".   Similarly,  if  the  loss  of 
2  bits  occurs  and  the  information  to  the  right  of  L"  is  L', 
L'L"  is  received  in  the  position  of  special  character,-  and  hence 
the  error  cannot  be  located. 

For  the  case  of  bit  loss  or  gain,  if  the  special  character 
was  of  length  ij.  and  the  information  following  it  was  a  dupli- 
cate of  special  character,  say 

L»   L" 
r 1   i r 

0001000111011 

and   a    loss    of  ij.  bits    occurs,    the  message   received  will  be 

L'  t    L"t 
000100011. 
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In  this  case  the  synchronizing  character  seems  to  be  un- 
disturbed whereas  the  loss  of  l\.   bits  has  occurred.   Therefore 
the  synchronizing  character  is  of  length  greater  than  or  equal 
to  2b  +  1  for  bit  loss  and  gain  of  1  bit,  and  greater  than  or 
equal  to  b  +  1  for  bit  loss  or  gain  of  1  bit. 

Error  Correcting  Codes  for  Use  with  Bit  Loss  and  Gain  Code 

The  choice  of  error  correcting  code  would  depend  on 
whether  the  channel  introduces  other  errors  besides  bit  loss 
and  gain  or  not,  such  as  additive  errors.   A  burst  error  cor- 
recting code  is  used,  in  case  the  transmission  channel  intro- 
duces even  bursts.   In  case  of  the  former,  the  special  charac- 
ter locates  the  approximate  position  of  error  and  the  only 
requirement  is  to  send  in  enough  redundancy  for  determining  the 
error  pattern.   Bits  of  redundancy  are  sufficient  for  the  cor- 
rection of  errors  caused  by  length  correction.   A  parity  check 
word  called  longitudinal  redundancy  check  (LRC)  is  formed  and 
is  used  for  error  detection.   When  the  information  is  received, 
length  correction  is  applied  and  its  location  in  the  block  is 
noted.   An  LRC  is  calculated  on  the  information  and  compared 
with  the  one  received  with  information.   A  binary  word  is  formed 
having  '1'  where  received  and  calculated  LRC  differ  and  '0' 
where  they  agree.   This  is  the  error  pattern. 

The  number  of  redundant  bits  on  a  message  with  k  informa- 
tion digits  is 

k/j(2L  +  1)  +  J  (32a) 

where  ( 2L  +  1)  is  the  length  of  the  synchronizing  character. 
In  this  equation  (2L  +  1)  will  always  be  odd,  i.e.,  1,  3,  $,    7, 


k.0 

.  ..,  and  since  the  expression  k/j(2L  +  1)  cannot  be  a  fraction 
of  a  bit,  this  will  be  equal  to  1  bit  which  means  there  will  be 
(j  +  1)  bits  of  redundancy.   The  redundant  bit  in  addition  to 
j  bits  of  redundancy  required  to  detect  the  error  pattern  is 
the  parity  check  on  the  information  digits. 

In  order  to  obtain  the  optimum  value  of  j  or  minimum  re- 
dundancy, Eq.32a  is  differentiated  and  equated  to  zero  as 
below. . 

d/dj[k/j(2L  +   1)    +    j]  =  0  (33) 

-k/j2(2L  +1)    +1=0 


j2  =  k(2L  +  1),  or  j  =/k(2L  +  1)  ( M 

This  gives  the  minimum  amount  of  redundancy.  For  in- 
stance, 100  information  digits  are  to  be  coded  into  a  code 
capable  of  correcting  bursts  of  up  to  ]+  bits. 

j  =  /k(2L  +  1)   =  /l00(2  x  k.  +   1)    =   ^900 
=  30  bits 
This  code  will  not  meet  the  objective  if  the  additive 
errors  are  not  in  the  vicinity  of  bit  loss  and  gain.   If  the 
additive  errors  occur  in  the  vicinity  of  bit  loss  and  gain, 
then  a  burst  error  correcting  code  must  be  used  such  as  Fire 
code  and  Hagelberger  code.   The  Fire  code  is  a  cyclic  burst 
error  correcting  code  while  Hagelberger  code  is  a  recurrent 
error  correcting  code.   Fire  codes  have  the  disadvantage  that 
these  require  a  k-stage  buffer  storage  at  the  receiver  to  hold 
the  message  to  be  corrected.   With  the  Hagelberger  code  this 
problem  is  eliminated  but  complexity  of  electronic  hardware 
required  is  directly  proportional  to  the  length  of  bursts  to 
be  corrected. 


kl 


In  order  to  correct  a  burst  of  b  or  fewer  bits  in  a  mes- 
sage with  k-inf ormation  digits,  Fire  code  has  (2b  -  1)  + 
log(n  +  1)  bits  of  redundancy  where  n  is  equal  to  k  plus  number 
of  bits  of  redundancy;  i.e., 

n  =  k  +  (2b  -  1)  +  log(n  +  1)  (35) 

A  good  approximation  for  the  value  of  j  may  be  obtained 
by  minimizing  the  numbers  of  redundancy  bits,  including 
special  characters.   Assume  that  it  is  desired  to  correct  all 
bits  of  length  j  -  1  or  less. 

0  =  d/dj[n/j(2L  +1)  +  2(j  -  1)  -  1  +  log(n  +  1)]  (36) 
Substituting     n  =  k  +  (2b  -  1)  +  log(n  +  1) 
and  b  =  ( j  -  1) 


d 
0  =  — 


"k  +  2(j-l)  -  1  +  log(n+l) 


=    [-k/j2   +   2/j2   +   l/j2   -    log(n+l)/j2 
-  [k  -   3   +   log(n  +   1)](2L  +   1) 


(2L+1)  +  2(j-l)  -  1 
+  log(n+l) 

— 

(2L+1)  +  2 


(37) 


,2 


+  2=0 


or   j  = 


[(k  -  3)  +  log(n  +  Dj(_2L  +  D_ 


(38) 


This  is  the  optimum  value  of  j  or  minimum  redundancy. 
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APPENDIX 


k3> 

Some  of  the  pertinent  definitions  and  rules  of  algebra  of 
rings  and  fields  (Peterson,  19&1)  as  used  in  the  study  of  Error 
Correcting  Codes  are  reviewed. 

A  group  is  a  system  with  one  operation  and  its  inverse, 
such  as  addition  and  subtraction,  or  multiplication  and  divi- 
sion.  A  group  obeys  the  laws  of  closure,  associativity,  and 
has  an  identity  element  as  well  as  inverse  of  each  element. 
A  group  which  satisfies  Commutative  law  is  called  an  Abelian 
group.   A  ring  has  two  operations  with  the  inverse  for  the 
first  operation  only.   A  ring  is  an  Abelian  group  under  addi- 
tion and  the  product  of  its  two  elements  is  defined.   It  obeys 
Associative  and  Distributive  laws.   A  ring  is  called  Commuta- 
tive if  its  multiplication  operation  is  Commutative,  i.e.,  for 
any  two  elements  "a"  and  "b",  ab  =  ba.   A  field  has  two  opera- 
tions both  with  inverses  and  is  a  Commutative  ring  with  a  unit 
element  in  which  every  nonzero  element  ha s  a  multiplication 
inverse. 

A  set  "V"  of  elements  is  a  vector  space  over  a  field  "F" 
if  it  satisfies  the  Distributive  and  Associative  laws.   It  is 
an  Abelian  group  under  addition  and  the  product  of  the  vector 
"V"  and  any  field  element  is  defined.   A  set  of  elements  is 
called  a  linear  associative  algebra  A,  it  is  a  vector  space  over 
field  F  and  for  any  two  elements  u  and  v  of  A,  product  uv  is 
defined  in  A.   It  also  satisfies  the  Associative  law  and  Bi- 
linear law,  for  any  three  elements  u,  v,  and  w  of  Z  (uv)w  = 
u(vw)  (Associative  law) .   If  c  and  d  are  scalars  in  F  and  u, 
v,  and  w  are  vectors  in  A,  then  u(cv  +  dw)  =  ( cuv  +  duw)  and 
( cv  +  dw)u  =  (cvu  +  dwu)  (Bilinear  law).   Row  space  of  an  n  x  m 


matrix  (aij)  Is  the  set  of  all  linear  combinations  of  row 
vectors  ar.d  the  dimension  of  the  row  space  is  called  the  row 
rank.   Similarly,  column  space  of  a  matrix  is  the  linear  com- 
bination of  column  vectors  of  a  matrix  and  Is  called  the  column 
rank. 


Example 


r 


Row  rank  =  3 
Column  rank  =  5 


10  0  11 

0  10  10 

0  0  10  1 

Elementary  row  operations  consist  of  interchange  of  any 
two  rows,  multiplication  of  any  row  by  a  nonzero  field  element, 
and  addition  of  any  multiple  of  any  row  to  another. 

Polynomials.   A  polynomial  f(x)  =  f q  +  f]_x  +  ...  +  fnxn 
is  of  degree  n.   It  is  called  monic  if  the  coefficient  of  high- 
est power  of  x  is  1.   A  polynomial  p(x)  of  degree  n  which  is 
not  divisable  by  any  polynomial  of  degree  less  than  n  but 
greater  than  zero  is  called  Irreducible .   The  greatest  common 
divisor  of  two  polynomials  is  their  common  polynomial  of  great- 
est degree.   Two  polynomials  are  said  to  be  relatively  prime 
if  their  greatest  common  divisor  is  1. 

Galois  Field.   If  p(x)  is  an  irreducible  polynomial  with 
coefficient  in  a  field  P,  then  the  algebra  of  polynomials  over 
F  modulo  p(x)  is  a  field.   The  original  field  is  called  a  ground 
field.   The  field  polynomials  over  GF(p)  modulo  an  irreducible 
polynomial  of  degree  m  is  called  the  Galois  Field  of  p^elements 
and  denoted  as  GFCp10)  .   For  any  number  q  =  pm   that  is  a  power 
o^  a  prime  number,  there  is  a  field  GF(q)  which  as  q  elements. 
T]   polynomial  has  as  root3  all  the  (q  -  1)  nonzero  elements  of 


GF(q)  and  the  polynomial  ( xm  -  1) is  divisible  by  (xn  -  1)  if 
and  only  if  m  is  divisible  by  n.   In  G-F(q)  there  is  a  primitive 
element,  i.e.,  an  element  of  order1  (q  -  1).   Even  a  nonzero 
element  can  be  expressed  as  a  power  of  «<,  that  is  multiplica- 
tive group  of  GF(q)  is  cyclic.   A  cyclic  group  consists  of  all 
the  powers  of  one  of  its  elements.   For  instance,  GF(2*+)  and 
its  representation  are  illustrated  in  Table  11. 

This  is  a  cyclic  operation  repeated  after  every  three  ele- 
ments and  note  that  addition  is  modulo  2  here. 
-<3  =  J 

A  =  -<°  +  *1 

£  =     .  ^   +  -<2 

J>  =  #   +  J 

jc  =  j^  +  j?  =  1  +  .-<  +  *,+  -<2 

li 

0 
=  1  +  -<2  and  so  on 

Every  polynomial  p(x)  of  degree  n  irreducible  over  GF(q) 
is  a  factor  of  ( xqm  -  m)  and  also  all  the  roots  of  an  irreduc- 
ible polynomial  are  of  the  same  order. 

Polynomial  Representation  of  Binary  Information 

Consider  a  code  n  long  in  which  k  are  information  digits 
and  remaining  n  -  k  are  parity  checks.   The  message  correspond- 
ing to  the  polynomial  ( 1  +  x  +  7?   +   x^)  is  110101.   These  poly- 
nomials obey  all  polynomial  laws  except  addition  which  is 


1^6 


Table  11 


Elements  of  GP(2^4-)  :  Representation 

-<n  =  1  (1000) 

(0100) 
(0010) 
(0001) 
(1100) 
(0110) 
(0011) 
(1101) 
(1010) 
(0101) 
(1110) 
(0111) 
(1111) 
(1011) 
(1001) 
(1000) 

modulo  2,  that  is 

iei  =  o®o  =  o 

o$i  =  ieo  =  i 

Addition,  multiplication,  and  division  of  these  poly- 
nomials are  given  below. 


■*!  = 

-< 

^  = 

^ 

■<3  = 

*3 

A  = 

l 

+ 

Jk 

+5  = 

-< 

+ 

*a 

6 
-<     = 

-<2 

+ 

J 

J  = 

l 

+ 

-< 

+ 

J 

J  = 

l 

+ 

jfi 

^  = 

-< 

+ 

J 

*io= 

l 

+ 

Js 

+ 

J? 

-*11- 

J, 

+ 

J 

+ 

A 

*12= 

l 

+ 

-< 

+ 

^ 

+ 

J 

^= 

l 

+ 

-<2 

+ 

J 

*U»- 

l 

+ 

J 

^15= 

l 

hi 


Addition 


1   +   x 


x  +   x< 


1    +    X '  +    X' 


+   x3   +   x^ 
+   x^ 


+  x1 


k 


+     XJ     + 


x^ 


Multiplication 
1  +  x 
1  +  x 


+  x3  +  rA 


1  +  x 


+  X 


3  +  x^ 


+  X  +  X 


+  y^   +  x^ 


2     3 
+  x  +  xJ 


+  x5 


Division 


1.x- 


1  •  x^  +  0  •  x  +  1 


l«x^  +  1-x^  +  1-x3  +  0-x2  +  l»x  +  0 

l-x£  +  Q.x^  +  l-x3 

1-x^  +  0-x3  +  0.x2  +  1-x  +  0 

1-xh  +   0.x3  +  l.x2 

0-x3  +  l«x2  +  1-x  +  0 
Q.x3  +  l»x2  +  Q.x 

1-x  +  1 


It  is  interesting  to  note  that  if  only  the  coefficient 
of  the  above  polynomial  is  divided,  the  result  is  the  binary 
representation,  as  follows. 


w 


110 


101 


111 

0    10 

10    1 

1   0 

0 

1   0 

1 

0 

110 

10    1 

1  1 


In  modulo  2  operations,  -1  =  +1  and  addition  and  subtraction 
are  equivalent. 
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In  a  communication  system  which  usually  consists  of  a 
source,  encoder,  channel,  decoder,  and  a  receiver,  an  encoder 
is  used  to  improve  the  efficiency  of  the  channel  by  increasing 
the  immunity  of  transmitted  message,  whereas  a  decoder  decodes 
the  received  signal  to  recover  the  original  message.   Communi- 
cation channels  are  either  noiseless  or  noisy,  such  as  binary 
symmetric,  binary  erasure,  and  cascaded  channel.   A  noiseless 
channel  has  one  and  only  one  nonzero  element  in  each  column  of 
its  channel  matrix.   A  deterministic  channel  has  one  and  only 
one  nonzero  element  in  each  row  of  its  channel  matrix.   This 
report  covers  binary  codes  for  memoryless  discrete  sources 
using  noiseless  and  noisy  channels. 

Binary  encoding  procedures  for  discrete  memoryless  sources 
and  the  noiseless  channels  by  Shannon-Fano,  Shannon,  Gilbert- 
Moore,  and  Huffman  are  not  devised  to  detect  or  correct  any 
errors  in  a  received  message  but  instead  to  optimize  the  coding 
efficiency.   Shannon-Pano  codes  are  100  per  cent  efficient  when 
the  probability  of  occurrence  of  each  message  Xk  is  of  the  form 

P(Xk)  =  2"nk 
where  n^  is  any  positive  integer.   Shannon's  binary  codes 
exist  for  such  sources  if  the  inequality 

t.     2_nk  ^  1 
k-1 

is  satisfied,  whereas  Gilbert-Moore  encoding  exists  for  source 

alphabets  which  satisfy  the  following  inequality: 

2!-nk  ^  P(xk)  sc  22"nk,  k  =  1,  2,  . . . ,  N 

Huffman's  minimum  redundancy  code  is  the  most  efficient  code 


for  a  specified  source,  where  minimum  redundancy  implies  an 
optimum  code  defined  as  a  code  with  minimum  average  word 
length  for  a  given  source  probability  matrix. 

Slepian  (1956)  generalized  the  error  detecting  and  cor- 
recting codes  introduced  by  Hamming  (1950)  and  Reed-Muller. 
These  codes  now  form  a  subclass  of  a  larger  class  of  codes 
called  Group  Codes.   Bose-Chaudhuri  codes  are  the  general  form 
of  all  these  codes  and  their  existence  theorem  states  that, 
"for  any  m  and  t,  there  exists  a  Bose-Chaudhuri  code  of  length 
2in"1  which  corrects  all  combinations  of  t  or  fewer  errors  and 
has  no  more  than  mt  parity  check  symbols"  (Peterson,  196l) . 
Reed-Muller  codes  cover  a  wide  range  of  rate  and  minimum  dis- 
tance.  Fire  codes  can  be  used  for  correcting  a  single  burst  of 
error  while  Reed-Solomon  code  is  capable  of  correcting  more  than 
one  burst  of  errors.-  The  average  length  of  the  latter  is 
shorter  than  that  of  the  former  for  correcting  the  same  length 
of  burst. 

Sellers  (1962)  introduced  a  block  code  that  can  correct  an 
error  caused  by  gain  or  loss  of  a  bit  within  the  block.   This 
code  is  constructed  by  inserting  a  special  character  into  a 
burst  error  correcting  code  at  periodic  intervals.   This  special 
character  locates  the  approximate  position  of  the  bit  loss  or 
gain.   Prom  that  location,  a  bit  is  inserted  or  removed  from 
the  block  depending  on  whether  a  loss  or  a  gain  has  occurred. 
The  error  correcting  code  then  corrects  the  erroneous  bits 
between  where  the  error  occurred  and  where  the  correction  took 


place.   This  code  can  be  generalized  to  correct  the  loss  or 
gain  of  a  burst  of  bits.   Since  no  bounds  have  been  derived, 
the  efficiency  of  this  method  is  not  known  as  compared  to 
other  possible  codes. 


